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DOCUMENT -IDENTIFIER: US 5537528 A 

TITLE : System and method for inputting scene information 



Abstract Text (1) : 

A scene information editor 12 extracts, for a plurality of scenes, still image data 
on the representative frames of scenes from a representative frame file, and 
arranges the still images along with a time axis and in chronological order along a 
time axis for display on the screen of a display . For the time period corresponding 
to the part of the time axis specified by a user, the editor extracts motion image 
data from an LD and displays the motion image data on a TV monitor. The scene 
information editor also extracts the information given to those scenes from a scene 
information file 21 and graphicall ^afsplays the~~inf ormatiom on the screen of the' 
display 14 at the same time. When the user inputs an edit command, the scene 
information editor executes it for the edit scene information file 21. 



Brief Summary Text (4) : 

Terms used in the specification are defined before the related art is described. It 
can be considered that a motion image consists of sequences of still frames . In the 
specification, an y frame sequence constituting a portion of the whole motion image 
is called a scene. Scene information is distinguished from the frame image itself of 
a scene, and denotes information provided by a user or application program on^ an 
individual scene. In particular, it includes the position of a scene in the motion 
image (i.e., the starting and ending frame numbers and the time code), the semantic 
contents of a scene (i.e., the keywords, attributes, and representative frame ) , the 
relationship between scenes (i.e., the identifier of parent or child), and 
information on scene changes (i.e., the position of a change point in the motion 
image, the change type, and the probability) . 

Brief Summary Text (6) : 

Generally, in a motion image, a plurality of continuous scenes, when assembled, have 
a meaning as a single scene on a higher level, and thus motion images have a 
structural characteristic that scenes constitute a hierarchy. In addition, they have 
the temporal characteristic that an item such as a character, object, or background 
that can be a retrieval key of a scene appears in consecutive frames . In Patent 
Application No. 4-21443, submitted previously by the present applicant (May 11, 
1994, Ser. No. 240,803 to T. Kaneko et al . , abandoned Jun. 30, 1994, which was a 
continuation of filed Oct. 13, 1992, Ser. No. 959,820 to Kaneko et al., abandoned 
May 11, 1994) , a motion image management system is disclosed in which, on the basis 
of such motion image characteristics, the original motion image is split into scenes 
of a shorter duration, and information on the hierarchy of scenes and descriptions 
of the semantic contents of scenes, or still images of representative frames of 
scenes, are prestored in a storage medium as index information, thereby allowing 
random retrieval of scenes . 

Brief Summary Text (7) : 

FIG. 1 shows the concept of motion image management in the above-mentioned related 
art. Motion image 2, consisting of many (for instance, 30 per second) frames fl to 
fn, is partitioned into a plurality of scenes 4 that are shorter in duration than 
the original motion image 2, as shown in FIG. 1 (a), according to physical changes • 
in the frames , such as cuts, changes of camera angle, or changes in the semantic 
contents. The partitioning of the individual scenes 4 is relative and arbitrary. For 
instance, a certain scene can be split into a collection of scenes of shorter 
duration, and conversely, a plurality of consecutive scenes can be merged and viewed 
as a single scene on a higher level. To describe the logical structure of scenes 
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based on such an inclusion relationship, a hierarchical tree 1 is created as shown 
in FIG. 1 (b) . The entire motion image 2 corresponds to the root node (R) 5 of the 
hierarchical tree 1, and the split and merged scenes 4 correspond to the 
intermediate node 6 and leaf node 7. The arcs 3 indicating the lower and upper 
adjacent relationships of nodes represent the parent-child relationship of nodes. 
For each scene corresponding to one node of the hierarchical tree 1, one or more 
frames (rf) representative of that scene or representative frames 8 are defined, and 
their still image data (representative frame image data) are generated. In each node 
of the hierarchical tree, attribute data (At) 9 such as a title or description 
acting as a retrieval key for a scene are stored along with a reference pointer to 
the representative frame (rf ) . 

Brief Summary Text (8) : 

As shown in FIG. 1 (a), to create this hierarchical tree, the system first 
automatically detects change points in frames fl to fn of the original motion image 
2 and splits the motion image 2 into minimum unit scenes (cut scenes) such as All 
and A12 in order to generate a one-depth tree structure. A user then appropriately 
merges adjacent cut scenes to form scenes whose contents are related, For instance, 
Al may be created from All and A12, thereby creating a multi-depth tree structure in 
a bottom-up fashion. Alternatively, as shown in FIG. 2, the stored original motion 
image 2 may be split according to the user's decision into arbitrary scenes such as 
A, B and C, and each scene may then be further repeatedly split into arbitrary 
scenes of shorter durations (for instance, A may be split into Al, A2, and A3), 
thereby creating a tree structure in a top-down fashion. In every case, the 
multi-depth tree structure 1 is created by editing (splitting and merging 
repeatedly) scenes according to their semantic contents. 

Brief Summary Text (9) : 

Scene retrieval is performed by, matching of the attribute data 9 of nodes (5, 6, 7) 
in the hierarchical tree 1 and node navigation along the arcs 3, using scene 
information- -in this case, the starting and ending frame numbers of each scene, the 
hierarchy, the attributes, and the representative frame image data and reference 
pointers thereto. That is, a retrieval condition is specified, which may be a scene 
attribute (At) or a condition for tracing the hierarchical tree, such as searching 
for a scene corresponding to the parent, child, or sibling node of the specified 
node in the hierarchical tree 1. The still images of representative frame 8 and 
attribute data 9 are displayed as a result of retrieval, and motion image data are 
accessed and played back for the scene 4 selected by the user from these still 
images . 

Brief Summary Text (10) : 

FIG. 3 shows the structure of the scene information file disclosed in the 
above-identified prior application. FIG. 3 (a) shows the structure of a first file 
for storing the attribute data of scenes, in which one record is assigned to each 
scene 4 acting as a node of the hierarchical tree, and its identifier 80, its 
starting frame number 81, and the ending frame number 82 are stored. Further, the 
values 83 of attributes (At 11, At 12 . . . ) describing the contents of the scene, 
the frame number 84 of its representative frame, and the reference pointer 85 of its 
representative 'frame of the still image file 86 are also stored in the same record. 
As identifier 80 of a scene, for instance, a value is assigned that uniquely 
identifies the scene, for example, on the basis of the pair of starting frame number 
81 and ending frame number 82. To specify the hierarchical relationship of scenes, a 
record in which the identifier 87 of a parent scene and the identifier 88 of a child 
scene are paired is stored in a second file, as shown in FIG. 3 (b) . 

Brief Summary Text (13) : 

The f: J- r st problem is related to a procedure for detecting scene change points and 
identifying the starting and ending frame numbers of scenes. One example of an 
approach for automatically detecting physical scene changes is a technique described 
in Ioka, M. , "A Method of Detecting Scene Changes in Moving Pictures," IBM TDB Vol. 
34, No. 10A, pp. 284-286, March 1992. Generally, methods for detecting scene changes 
include comparison of the degree of change in the signal level between continuous 
frames, or the change in pixel value, with a threshold value. Because of this, the 
accuracy of such methods depends on the value to which the threshold is preset. If 
the threshold is set too low, the rate of detection failure (failure to detect a 
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scene change point) decreases, but the rate of erroneous detection (deeming a point 
other than a scene change point to be a scene change point) increases; if the 
threshold is set too high, the result is the opposite. Usually, it is difficult to 
set the threshold so that there are no detection failure or erroneous detections. 
Accordingly, scene change points must be verified and corrected by the user while he 
or she is actually viewing the motion image. Nevertheless, no user interface for 
efficiently verifying and correcting scene change points has been described up to 
the present. Because of this, it is cumbersome for the user to instruct the system 
to play back or stop the motion image, for example, and mistakes are easily made. 

Brief Summary Text (14) : 

The second problem is related to the procedure for describing a scene's contents. 
Although a text editor or the like is normally used to create a file describing the 
scene contents, no efficient data input procedure has yet been proposed. Because of 
this, users sometimes create unnecessary work files and work areas to write 
essentially the same scene information into different files. In addition, updating 
one file requires a cumbersome procedure such as a check by the user of the updated 
file's consistency with correlated files. Furthermore, since correlated information 
cannot be referred to for scene description, redundant operations such as repeated 
playback of the same scene and repeated input of the same frame number are required. 

Brief Summary Text (15) : 

The third problem is related to the efficiency of the input operation in general. In 
the past, the user had to directly input character and numeric data such as frame 
numbers from an input device such as a keyboard, which was cumbersome. In addition, 
it was difficult to effectively feed back input scene information to find and 
correct input errors immediately. 

Brief Summary Text (16) : 

In Ueda, H. et al . , "A Proposal of an Interactive Video Image Editing Method Using 
Recognition Technology, " Proceedings of the Institute of Electronics and 
Communication Information Engineers, D-li, Vol. J75-D-II, No. 2, pp. 216-225, Feb. 
1992, a technique is disclosed for automatically splitting an original motion image 
into scenes, and displaying the image of the leading frame of each scene on a 
display device in order to browse scenes that are to be subjected to editing 
(authoring) . A software product having the brand name of VideoShop announced by DiVA 
Corporation and introduced in MACLIFE No. 45, May 1992, pp. 242-245 also provides a 
function for selecting scenes and arranging them in a desired sequence. On its 
editing screen, a new sequence of scenes into which the original sequence of scenes 
has been rearranged is displayed along with a time axis. However, these techniques 
are intended to enable the user to select and rearrange scenes in order to create a 
motion image that is different from the original one; they are not directed to 
making the scene information input efficient. 

Brief Summary Text (24) : 

A scene information editor extracts, for a plurality of scenes, still image data on 
the representative frames of scenes from a representative frame file, and arranges 
the still images along with a time axis and in chronological order along a time axis 
for display on the screen of a display . For the time period corresponding to the 
part of the time axis specified by a user, the editor extracts motion image data 
from an LD and displays the motion image data on a TV monitor. The scene information 
editor also extracts the information given to those scenes from a scene information 
file and graphically displays the information on the screen of the display at the 
same time. When the user inputs an edit command, the scene information editor 
executes it for the edit scene information file. 

Drawing Description Text (13) : 

FIG. 12 is a diagram illustrating the basic construction of the display panel of the 
scene information editor; 

Drawing Description Text (14) : 

FIG. 13 is a diagram showing an example of a representative frame display area in 
the scene information editor display panel; 
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Drawing Description Text (15) : 

FIG. 14 is a diagram showing an example of the t ime bar display area in the scene 
information editor display panel; 

Drawing Description Text (18) : 

FIG. 17 is a diagram showing a panel display example in the representative frame 
change mode of the scene editor; 

Drawing Description Text (19) : 

FIG. 18 is a diagram showing a representative frame change and its propagation; 
Drawing Description Text (20) : 

FIG. 19 is a diagram showing a panel display example of the scene hierarchy editor; 
Detailed Description Text (2) : 

FIG. 4 shows the flow of a process from splitting a motion image consisting of 
chronologically ordered frame sequences into scenes that are partial frame sequences 
of the whole motion image, through inputting scene information after verification 
and correction of the split ted scenes, up to registering the input information in a 
motion image database. The invention included in this application includes (a) a 
step for detecting scene change points, (b) a step for generating scene information 
on the basis of the detected scene change points and storing it in a scene 
information file, (c) a step for graphically displaying scene information on a 
display, (d) a step for executing the edit command specified by the user operating 
the input means to edit the scene information file, and (e) a step for registering 
the scene information contained in the scene information file in the database means. 



Detailed Description Text (4) : 

The above step (b) includes a step for creating a representative frame file in which 
still image data on representative frames are stored for scenes defined by logically 
splitting the motion image according to the scene change information; a step for 
creating a scene information file that contains the starting and ending frame 
numbers of scenes; the type and probability of each scene change; the representative 
frame numbers; and the addresses at which the data on representative frames are 
stored. 

Detailed Description Text (5) : 

In the above step (c) , one or more charts graphically presenting scene information 
are displayed . The possible types of charts are the scene chart 30, scene hierarchy 
chart 31, and scene keyword chart 32, examples of which are shown in FIGS. 5 to 7 . 
The type of chart to be displayed is selected by the user. 

Detailed Description Text (6) : 

The scene chart 30 exemplified in FIG. 5 displays the shrunken images 33 of 
representative frames selected for individual scenes, the shrunken images 3 3 being 
arranged in time sequence along the time bar 34, which is an object representing a 
time axis. This chart has a region 35 that displays the type and probability of a 
scene change, at a scene change point, or the leading frame of a scene. 

Detailed Description Text (7) : 

The scene hierarchy chart 31 exemplified in FIG. 6 is displayed simultaneously with 
the scene chart 30, and represents the hierarchy depth of scenes according to their 
distance from the time bar 34. Specifically, boxes 36 are arranged in a direction 
perpendicular to the time bar 34, and the order of the boxes is made to correspond 
to the hierarchy depth. An object representing information on the hierarchy of a 
scene is displayed in an area 37 whose position is defined by the interval 
corresponding to the particular scene on the time bar 34 and the box 36. 

Detailed Description Text (8) : 

In the scene keyword chart 32 exemplified in FIG. 7, the boxes 38 are arranged in a 
direction perpendicular to the time bar 34 of the scene chart 30, and if a keyword 
is assigned to a scene, an object representing its information is displayed in the 
area 39, whose position is defined by the interval corresponding to the particular 
scene and the box 38 of the particular keyword. Preferably, the keyword is 
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associated with a still image such as an icon symbolizing its contents, and is 
displayed by using the still image. Alternatively, by associating a still image with 
the identifier of a keyword, the still image itself is treated in the same manner as 
the keyword. In this specification, the keyword is a concept that includes both the 
identifier and the still image. 

Detailed Description Text (9) : 

In the above step (d) , an edit command is accepted. This edit command is specified 
by the user through direct manipulation such as pointing to, moving, and copying an 
object displayed on the screen in the above step (c) . This step includes a sub-step 
of responding to the specification of the shrunken image 33 of the representative 
frame by the user in each of the above charts to play back the scene corresponding 
to the particular representative frame, and if an interval on the time bar 34 is 
specified, playing back the motion image corresponding to the specified interval. 

Detailed Description Text (10) : 

If the scene chart is selected in the above step (c) , a command for adding or 
deleting a scene change point is accepted in one mode, a command for changing a 
representative frame is accepted in a second mode, and a command for updating scene 
attribute information is accepted in a third mode. In this way the scene information 
file is edited, and sometimes the representative frame is also edited. If the scene 
hierarchy chart is selected, commands for splitting and merging scenes and a command 
for changing the hierarchy depth are accepted, and as a result the scene information 
file is edited: Further, if the scene keyword chart is selected, commands for 
setting and canceling a keyword allocation to a scene are accepted and used to edit 
the scene information file. 

Detailed Description Text (11) : 

Alternatively, if both the scene hierarchy chart and the scene keyword chart are 
selected at the same time in the above step (c) , both charts are simultaneously 
displayed . In the first mode, in this case, in response to manipulation of an object 
on one chart, the scene information corresponding to that chart is updated. In the 
second mode, in response to manipulation of an object on one chart, the scene 
information corresponding to the other chart is updated. In the third mode, in 
response to manipulation of an object on one chart, the sets of scene information 
corresponding to all the charts are simultaneously updated. 

Detailed Description Text (12) : 

When the above step (d) is repeated and the end of the input operation is specified, 
in the above step (e) , a command for registering the scene information file and, 
when appropriate, a command for registering the file of representative frame images 
are generated and sent to the database means. 

Detailed Description Text (13) : 

FIG. 8 shows an example of the configuration of a scene information input system in 
which the present invention is incorporated. The system consists of a motion image 
storage and display unit 10, a scene change detection unit 11, a scene information 
editor 12, a registration processing unit 13, a display 14, and an input device 15. 

Detailed Description Text (14) : 

The motion image storage and display unit 10 includes a laser disk (LD) 16 on which 
a motion image is stored as an analogue image, a laser disk player (LD player) 17, a 
TV monitor 18, and a video signal converter 19. The LD player 17 has an interface 
that responds to an external control signal from the scene information editor 12 or 
the like to send the analogue image in the LD 16 to the TV monitor 18 or video 
signal converter 19. The LD player 17 also includes an interface that responds to an 
external control signal by returning data such as the number of the frame that is 
currently being played back. The analogue image is input from the LD player 17 to 
the video signal converter 19, which converts analogue signals into digital ones to 
output a digital image. 

Detailed Description Text (15) : 

The scene change detection unit 11 outputs a control signal to the LD player 17 to 
sequentially capture the image data of consecutive frames, detect scene change 
points, and calculate scene change probabilities. As a result of detection, a scene 
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information file 20 is created in which are recorded, the number of each frame in 
which a scene change has occurred, the type and probability of the change^ and the 
number of the frame representative of the scene (for instance, the leading frame) . 
FIG. 9 shows an example of a scene change information file 20, in which a change 
frame number 901, a representative frame number 902, a change probability 903, and a 
change type 904 are stored as one record. Of the values in the field 904, n i" 
represents "normal" and "2" represents "dissolve." What they mean is described 
later, along with the details of the scene change detection processing. 

Detailed Description Text (16) : 

The scene information editor 12 consists of a scene editor module, a scene hierarchy 
editor module, a scene keyword editor module, and a cross editor module. Every 
module displays scene information stored in a scene information file 21 and 
representative frame file 22 on the display 14, and responds to a user's 
specification through the input means 15 to update the scene information file 21 and 
representative frame file 22. In particular, the scene editor inputs the scene 
change information file 20 to create a scene information file 21 and representative 
frame file 22 . 

Detailed Description Text (17) : 

FIG. 10 shows the structures of the scene information file 21 and representative 
frame file 22. The scene information file 21 consists of files (a), (b) , and (c) , 
but files (a) and (b) are same as those shown in FIG. 3. Accordingly, the same 
elements are referred to by the same numerals, and description of files (a) and (b) 
is omitted. In this embodiment, because of the construction of a motion image 
database in which scene retrieval is performed by means of keywords such as the 
names of characters or the types of background, an identifier 89, starting frame 
number 91, ending frame number 92 and keyword 90 assigned thereto are stored as one 
record in the third file (c) for each scene. Examples of such motion image databases 
are disclosed in Oomoto, E. and Tanaka, K. , "A Video Object Model and Its 
Application to a Video Database System," IEICE Tech. Rep. Vol. 89, No. 466, 1990, 
pp. 33-40 and patent application No. 4-65620 (U.S. Pat. No. 5,428,774 to Hashihara 
et al.) . The modules of the scene information editor 12 identify a record by means 
of the scene identifiers 80, 87, 88, and 89 of files (a), (b) , and (c) , and refer to 
and update scene information. Although it is not shown, file (a) may include scene 
change type and probability fields. The representative frame file 22 is a collection 
of still image files 86 created for individual scenes. 

Detailed Description Text (18) : 

In response to instructions issued by the user through the input device 15, the 
scene information editor 12 issues control signals to specify pauses in the playback 
of an image and frame number acquisition. In response thereto, the LD player 17 
stops the playback of the image and displays the current still frame on the TV 
monitor 18 while supplying the number of the still frame to the scene information 
editor 12. The input device 15 is typically a character and numeric value input 
device such as a keyboard, mouse, or touch panel. 

Detailed Description Text (19) : 

The registration processing unit 13 generates a command for storing the scene 
information file 21 and, when appropriate, a command for storing the representative 
frame file 22 in the database 24, and sends them to the database management unit 23. 



Detailed Description Text (22) : 

The scene change information file 20, scene information file 21, representative 
frame file 22, keyword information file 25, and keyword image file 26 are usually 
stored in the main memory or an external storage device of a computer. As shown in 
FIG. 10, these files may be a collection of sub-files. 

Detailed Description Text (24) : 

The scene change detection unit 11 is now described in detail. This unit calculates, 
by a method depending on the type of change, the degree of change for each frame in 
order to detect scene change points of different types. The degree of change is the 
change in the pixel value from the preceding frame . 
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Detailed Description Text (25) : 

The most representative examples of scene change types are "normal" and "dissolve," 
and a method for detecting them is described below. Here, "normal" means that the 
camera angle completely changes from one frame to the next, or that a scene changes 
on account of the passage of time between events in a motion image. "Dissolve" is a 
scene change caused by simultaneous occurrence of fade-in and fade-out. A normal 
scene change ends immediately, whereas a dissolving scene change requires several 
dozen frames . 

Detailed Description Text (26) : 

The details of an example of a process for detecting the "normal" type of scene 
change are disclosed in the references given in the description of the related art. 
In brief, for each frame, the frame difference mean FDM(j), which is the absolute 
differences in pixel value between the current frame (j) and the preceding frame 
(j-1) is, first calculated, and then the difference DFDM(j) between the mean value 
FDM(j-l), which is the difference between the frame before last (j-2) and the 
preceding frame (j-1), and the previously calculated mean value FDM(j) of the 
difference is calculated. Subsequently, a check is made to determine whether a zero 
cross has occurred between the calculated DFDM(j) and the last DFDM(i-l) . If a zero 
cross has occurred, the absolute value of the difference between DFDM(j) and 
DFDM(j-l) is compared with a preset threshold. If the difference is larger than the 
threshold, frame number j is recorded in the scene change information file 2 0 as the 
change frame number 901 (FIG. 9), and it is deemed that a normal type of scene 
change has occurred. A value representing "normal" is set in the field 904. As the 
representative frame number 902, the value resulting from the addition of a 
predetermined value to the change frame number 901 is recorded in the scene change' 
information file 20. 

Detailed Description Text (28) : 

1) The image of one frame is captured from the video signal converter 19. 
Detailed Description Text (29) : 

2) If there is no frame any more, the processing is terminated. 
Detailed Description Text (32) : 

5) The difference between the current frame and the preceding frame is calculated 
for each sub-sampled pixel. 

Detailed Description Text (34) : 

6) The frame difference calculated at this time is divided by the previously 
calculated frame difference at each sub-sampled pixel. Further, the ratio of all the 
sub- sampled pixels whose calculated ratio show positive values is calculated. 

Detailed Description Text (36) : 

8) Whether the current frame is located at local maximum or minimum in the series of 
the filtered ratio is checked. If it is local maximum, the ratio is recorded as 
Lmax, and if local minimum, as Lmin. If the current frame is neither maximum nor 
minimum, the process returns to (1) . 

Detailed Description Text (38) : 

10) If the current frame is local maximum, the process advances to (11) . If minimum, 
the process skips to (12) . 

Detailed Description Text (39) : 

11) Since the current frame may be the starting point of dissolve, a value 
representing ON is set for a variable search switch, and the number of the current 
frame (assumed to be ST) is recorded. 

Detailed Description Text (40) : 

12) If the variable search switch is ON, the current frame may be the ending point 
of "dissolve," and thus the process moves to (13); If the switch is not ON, the 
process returns to (1) . 



Detailed Description Text (41) : 

13) If the difference between the number of the current frame (assumed to be CT) and 
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the number (ST) of the frame at which the search was started is within the search 
range, dissolve has happened. If the difference is outside the search range, 
dissolve has not happened. The search range is provided as a parameter at the 
starting point of the whole process for detecting scene changes. Usually, it is a 
value corresponding to a time of about three seconds. 

Detailed Description Text (43) : 

15) Since the type of change is "dissolve," a frame representing minimum is recorded 
as the ending point of the dissolve and the process returns to (1) . 

Detailed Description Text (44) : 

The frame number ST is recorded in the change information file 2 0 as the change 
frame number 901. A value representing "dissolve" is set in the field 904. The value 
resulting from adding a predetermined value to the change frame number 901 is 
recorded in the scene change information file 2 0 as the representative frame number 
902. The value obtained by normalizing the difference of the maximum and minimum 
calculated in step (9) for the starting frame of the dissolve is used as the scene 
change degree C. 

Detailed Description Text (46) : 

The scene information editor 12 is the main portion of the present invention. The 
method. of displaying scene information is first described below, and then the input 
operation procedure and the operations of all the editors are described. 

Detailed Description Text (47) : 

As shown in FIG. 12, the scene information editor 12 places on the screen of the 
display 14 a frame information display area 40, a processing mode display area 41, a 
video control button 42, a representative frame display area 43, a time bar display 
area 44, an edit area 45, and edit command area 46, and a display panel/manipulation 
mode switch area 47. The scene chart 30 shown in FIGS. 5 to 7 consists of the 
representative frame display area 43 and the time bar display area 44. In addition, 
the scene hierarchy chart 31 and scene keyword chart 32 are displayed in the edit 
area 45. 

Detailed Description Text (48) : 

The representative frame display area 43 is an area in which, for a plurality of 
scenes, as exemplified in FIG. 13, the shrunken images 33 of the representative 
frames are shown, and the scene change type 51 and probability 52 are displayed for 
each scene change point. Finally, the representative frame display page 
forward/backward button 53 is an area in which the scrolling of representative 
frames is specified. 

Detailed Description Text (49) : 

Since one scene has one representative frame if scene information is edited for cut 
scenes, the shrunken images 3 3 are placed in such a way that there is one for each 
scene. FIG. 13 shows an example of such placement. However, at higher levels in a 
hierarchy tree, one scene may have a plurality of representative frames . In this 
case, all of the shrunken images 33 of the particular scene may be shown in a time 
sequence. Alternatively, if the earliest representative frame, for instance, is 
automatically selected, the external appearance of the representative frame display 
area 43 is as shown in FIG. 13, even for higher level scenes. 

Detailed Description Text (50) : 

The time bar display area 44 is an area in which, as shown in FIG. 14, the cursors 
55, 56, and 57 are displayed on the time bar 34 representing a time axis. The work 
frame cursor 55 designates a frame, and the frame interval cursor 56 designates the 
starting and ending positions of a frame interval. The cursors 55 and 56 allow the 
user to set a frame or frame interval as an object to be manipulated in the step (d) 
described in "Means for solving the problems . " These cursors are moved interactively 
by the user operating input means 15 by, for instance, dragging a mouse cursor (not 
shown) . By clicking the time bar 34 with the mouse cursor, the state in which the 
cursors 55 and 56 are displayed and the state in which they are not displayed can be 
switched. The display frame cursor 57 indicates the current frame position of the 
motion image being displayed on the TV monitor 18. 
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Detailed Description Text (51) : 

In FIG. 14, an interval of an equal length (LO) is assigned to every scene. The 
number of frames corresponding to the length L0 depends on the scene. The positions 
of the cursors 55, 56, and 57 and the numbers of the frames designated by them are 
correlated as follows. That is, taking scene I in FIG. 14 as an example, the frame 
number F designated by the cursor 55 is calculated by the following expression, if 
it is supposed that the distance between the starting point of the scene and the 
point designated by the cursor 55 on the time bar 34 is LC and the starting and 
ending frame numbers of scene I are Si and Ei, respectively. 

Detailed Description Text (52) : 

The starting and ending frame numbers of the frame interval designated by cursor 56 
are calculated in a similar manner. In addition, the position at which cursor 57 is 
displayed is obtained by converting the number (F) of a frame that is being played 
back into a position (LC) on the time bar 34. 

Detailed Description Text (53) : 

In all the editors and modes described below, in response to specification of the 
shrunken image 33 of a representative frame by the user operating the input device 
15, or setting of a frame interval in the time bar display 44 with subsequent 
operation of the video control button 42, the motion image of a specified time 
period is played back. For instance, in FIG. 14, if the shrunken image 3 3 of scene I 
is clicked, the part of the motion image from frame Si to frame Ei is displayed on 
the TV monitor 18. In addition, when specified by the frame interval cursor 56, the 
part of the motion image from frame SS to frame EE is displayed on the TV monitor 
18. Alternatively, a window may be provided in part of the scene information display 
panel for displaying the motion image, or a scene may be played back in area 
occupied by the corresponding shrunken image 33. 

Detailed Description Text (54) : 

In FIG. 12, the frame information display area 4 0 shows information on a frame to be 
manipulated, such as the position of the said frame in the whole original motion 
image. The display panel/manipulation mode switch area 47 has areas for specifying 
display panel switching and selection of the processing mode in each panel . 
(including a mode switch button 58 and an editor switch button 59 shown in FIG. 15) , 
and an area for specifying the start-up of the registration processing unit 23 (the 
registration button 68 in FIG. 15) . The edit mode selected there is displayed in the 
processing mode display area 41, while the edit commands that can be selected in the 
current processing mode are displayed as a menu in the edit command area 46. The 
video control button 42 is an area for instructing the scene information editor 12 
to send a control signal for image playback to the motion image storage and display 
unit 10. In edit area 45, the scene information editor 12 displays scene information 
in the form of a scene chart, scene hierarchy chart, or scene keyword chart. The 
user specifies an edit operation by specifying, moving, and copying of the object 
displayed in the chart, setting the cursor in the time bar, and selecting of an edit 
command, and the scene information editor 12 executes the specified command. 

Detailed Description Text (57) : 

The scene editor allows the user to verify a scene change point and correct any 
error made by the scene change point detection unit 11 while viewing the 
representative frame of a scene or the motion image corresponding to the scene, to 
edit the attribute information of the scene and to change the representative frame . 

Detailed Description Text (58) : 

First, the scene editor reads the scene change information file 20, sends the frame 
number of the representative frame to the motion image storage and display unit 10, 
captures the image data of that frame from the motion image storage and display unit 
10, and sub-samples pixels, thereby creating and adding the still image file 86 to 
the representative frame file 22. Then, it extracts the frame number in which a 
scene change has occurred from the scene change information file 20, and calculates 
the starting and ending frame numbers of the scene from the frame number to create a 
scene information file 21 as shown in FIG. 10. 

Detailed Description Text (59) : 

Then, the scene editor accesses the scene information file 21, as exemplified in 
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FIGS. 15 to 17, and according to the data stored therein, presents scene information 
by using the scene chart 30. According to the user's specification, the scene editor 
displays either representative frames for all the scenes partitioned by the scene 
change detection unit 11 in chronological order, or only the scenes including scene 
change points whose probabilities are included in a previously specified range, 
including the preceding scene when appropriate. In the latter case, all the scenes 
meeting the condition are displayed either in chronological order or in the 
descending order of the possibility Q of detection error, according to the user's 
specification. The possibility Q of detection error is given, for instance, by 
. vertline . 0 . 5-P . vertline . . P is the probability of a scene change point. 
Incidentally, a scene change type may be specified in lieu of probability as the 
condition for selecting a scene, or both may be specified at the same time. 

Detailed Description Text (60) : 

The shrunken image 33 to be displayed is created by shrinking the image of the 
representative frame image 86. Because of the cost of using the storage device, and 
the quality and display speed of the image playback, the image size of one frame to 
be stored in the representative frame file 22 should preferably be between the 
original frame size and the size of the shrunken image 33. 

Detailed Description Text (61) : 

It has already been stated that a representative frame page forward/backward button 
53 is provided to cause display of a desired representative frame . In addition, in 
the panel shown in FIG. 15, when the endmost, lower-right-corner button 53 is 
clicked, the representative frames for one panel; in this example, for 48 scenes, 
are scrolled. Further, a command to allow the user to jump to any scene whose 
identifier is typed in the edit command area 46. 

Detailed Description Text (62) : 

The scene editor operates in modes for verifying and correcting scene change points, 
for editing the scene attributes, and for changing representative frames . If the 
work frame cursor 55 is displayed, the panels exemplified in FIGS. 15 to 17 are 
displayed in response to the user's selection of the mode switch button 58 to change 
the mode by using a mouse cursor (not shown) . If the work frame cursor 55 is not 
displayed, a target scene is specified by the user when the mode changes from scene 
verification and correction (FIG. 15) to scene attribute edit (FIG. 16) or 
representative frame change (FIG. 17). The specification is provided by clicking one 
of the shrunken images 33 in the scene change point verification and correction 
mode. If the specification provided by the cursor 55 conflicts with that provided by 
the mouse cursor, the predetermined specification of higher priority is used. 

Detailed Description Text (65) : 

To add a scene change point, the user uses the input device 15 to instruct that the 
motion image should be frozen at a frame in which the scene has changed, and 
subsequently selects the command for adding a scene change point, shown in the edit 
command area 46. In response to this, the scene editor splits the scene, with 
specified frame as the scene change point. Now, let us suppose that scene T is split 
into scenes Tl and T2 by an operation for adding a scene change point, and that 
scene Tl is earlier in time. If the representative frame of scene T is not included 
in scene Tl, for instance, the leading frame is treated as the default 
representative frame, and the image file 86 for that frame is created. The still 
image file 86 of scene T directly becomes the image file of scene T2 . Conversely, if 
the representative frame of scene T is not included in scene T2, the leading frame, 
or the frame frozen at the user's instruction, is treated as the default 
representative frame, and a still image file 86 for that frame is created. The still 
image file 86 of scene T directly becomes the image file of scene Tl . In either 
case, the records of scenes Tl and T2 are inserted into parts (a) and (c) of the 
scene information file 21 shown in FIG. 10, the record of the original scene T is 
deleted, and the scene chart 30 is updated on the basis of part (a) of the updated 
scene information file 21. 

Detailed Description Text (66) : 

To cancel a scene change point, the user specifies the extent of scenes merged into 
one scene by clicking a plurality of continuous scenes or by specifying the initial 
and final scenes with the frame interval cursor 56. Subsequently the user specifies 
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the representative frame of the new scene. This is done by clicking the shrunken 
image 33 of the representative frame of any of the scenes to be merged, or by 
freezing the video image displayed on the monitor 18 at a desired frame, and 
subsequently selecting the command for canceling scene change points, shown in the 
edit command area 46. In response to these specifications, the scene editor merges 
the specified scenes into a single scene, and the specified frame is defined as its 
representative frame . If no representative frame is specified, the earliest of the 
representative frames included in the interval for which merging is performed is 
selected. In this case, the record of the new scene is inserted into part (a) of the 
scene information file 21, and the records of the old, merged scenes are deleted, 
while the scene chart 3 0 is updated according to the contents of part (a) of the 
updated scene information file 21. 

Detailed Description Text (67) : 

Thus, in the scene change point verification and correction mode, the precision with 
which scene change points are detected can be substantially increased by setting the 
range of the degree of change within which detection failure and erroneous detection 
occur, and by having the user reobserve the frames preceding and following a scene 
change point that is included in the range. Furthermore, limiting the range makes 
the amount of labor required much less than that needed to observe all the scene 
change points. In addition, controlling the motion image display speed according to 
the probability of scene change reduces the number of complicated display control 
operations by the user, and thus makes the verification and correction work more 
efficient. 

Detailed Description Text (68) : 

The scene attribute edit mode is now described. In this mode, the scene attribute 
edit panel exemplified in FIG. 16 is displayed. The edit area 45 of this panel 
consists of a scene hierarchy display area 60 and a scene attribute input area 61. 
By using the cursor 55 to designate a representative frame in the scene chart 30, or 
by designating a node of the scene hierarchy tree displayed in the scene hierarchy 
display area 60, the user specifies a scene, whose attributes he or she wishes to 
describe, subsequently keys in scene attributes such as a title and a description of 
the contents, in the scene attribute input area 61, and selects an attribute 
recording command in the edit command area 46. The scene editor executes this 
command to store attribute data on the designated scene in the attribute field of 
the record maintained in part (c) of the scene information file 21. 

Detailed Description Text (69) : 

Finally, the representative frame change mode is described. In this mode, the 
attribute and representative frame change panel exemplified in FIG. 17 is displayed. 
The edit area 45 of this panel includes a candidate image display area* 62 , capture 
button 63, and registration button 64. In this mode, the user specifies that the 
representative frame of a scene should be changed to another frame of the scene. The 
specification of operation includes issuance of instructions for freezing the motion 
image on the monitor 18, and subsequent clicking of the frame capture button 63. In 
response to the specification, the scene editor inputs a video image from the motion 
image storage and display unit 10, creates a still image file of the specified 
frame, and displays its image in the candidate, image display area 62. In response to 
the subsequent clicking of the registration button 64, the scene editor adds the 
still image file 86 of a new representative frame to the representative frame file 
22, updates part (a) of the scene information file 21 in order to update the 
representative frame number 84 of the particular scene, and displays the still image 
of the new representative frame at the corresponding position in the representative 
frame display area 30. 

Detailed Description Text (70) : 

A change of the representative frame of a scene is propagated to the scene's parent 
and child scenes in the scene hierarchy. The example in FIG. 18 shows the states 
before and after a change of the representative frame of a scene that has child 
scenes. When the representative frame of scene A is changed from fl to f2 (<f 1) , the 
representative frame of child scene Al of A is also changed to f2, because Al 
includes frame f 2 . Accordingly, the pointer 191 of Al to the image file points to 
the same address as the pointer 190 of A. The change of the representative frame of 
the parent scene is similarly propagated to child scenes as far as leaf node scenes. 
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Simultaneously, the change of the representative frame of scene A is propagated to 
its parent scene if the representative frame of a scene that is the parent of A is 
included in A. The change of the representative frame of the child scene is 
similarly propagated to parent scenes as far as the root node scene. 

Detailed Description Text (72) : 

If the user presses the editor switch button 59 in the panels shown in FIGS. 15 to 
17, the scene editor is initiated, and as exemplified in FIG. 19, the scene chart 30 
is displayed in an area in the upper portion of the panels and the hierarchy chart 
31 showing scene hierarchy information is displayed in another area, in the lower 
portion. By default, all the scenes are subjected to display, but the user may 
previously limit displayed scenes to those are to become leaves or scenes of a 
specific hierarchy depth. In FIG. 19, only scenes that become leaves are displayed. 
In the initial state of the hierarchy chart, one depth is shown, with the whole 
motion image viewed as a parent scene and with each scene split by the scene change 
detection unit 11 viewed as a child scene. From this initial state hierarchy, the 
user inputs commands for changes in the scene hierarchy, such as splitting and 
merging of input scenes to create a complicated hierarchy for the scenes and they 
are executed. FIG. 2 0 shows the hierarchical relationship corresponding to FIG. 19 
in a manner similar to FIG. 2. The scenes A to L for which the shrunken image 33 of 
the representative frame is shown in FIG. 19 correspond to scenes SI, S2, S31, S32, 
S331, S3321, S3322, S333, S334, S3351, S336, and S337, respectively. 

Detailed Description Text (73) : 

A scene hierarchy change command is input directly by the user manipulating an 
object on the display panel. Specifically, the user performs the following 
operation. First, he specifies a scene to be manipulated by setting a cursor 55 or 
56 on the time bar 34 or by clicking the shrunken image 33. Subsequently, he 
performs one of the following operations. 

Detailed Description Text (76) : 

c) A split command is selected in the edit command area 46. In this case, the split 
point is indicated by the work frame cursor 55 on the time bar 34. 

Detailed Description Text (81) : 

The representative frames of child scenes that are newly created when a scene is 
split are automatically established on the basis of the representative frame of the 
parent scene. The example shown in FIG. 22 shows the states before and after the 
establishment of the representative numbers of child scenes Al and A2 and pointers 
to the image file, when scene A having frame fl as a representative frame is split 
at frame f2 (<fl) to create child scenes Al and A2 . The representative frame of A is 
fl, which has a reference pointer 220. Since the frame f2 at which splitting is 
performed has a frame number lower than fl, the representative frame of Al is the 
leading frame f 3 , which is a default, and the representative frame of A2 becomes fl, 
which is also the representative frame of A. Further, the pointer 221 for A2 to the 
image file points to the same address as the pointer 222 for A. There is an empty 
pointer 223 for Al, but, when the image file of frame' f3 is created, it is replaced 
by the address of the file. Conversely, if scenes Al and A2 are merged into A, 
whichever of the representative frames of Al and A2 is selected by the user becomes 
the representative frame rf of A. 

Detailed Description Text (85) : 

a) A frame interval or scene is specified in the scene chart 30. Subsequently, a box 
38 representing a desired keyword is clicked. The editor interprets this as a cancel 
command for keyword assignment if the specified keyword has already been assigned to 
the specified scene; otherwise it interprets it as a keyword assign command. 

Detailed Description Text (87) : 

c) Two frame intervals are established in the scene chart 30. Subsequently, in the 
edit command area 46, a command is selected for cutting the keyword bars 66 out of 
one interval and pasting them into the other. 

Detailed Description Text (92) : 

The files that the scene keyword editor edits in this mode are the keyword 
information file 25 and the representative file 26, both shown in FIG. 8. In the 
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keyword information file 25, the identifier, attribute, and storage address in the 
still image file are stored in one record for each keyword. The keyword image file 
26 is a collection of still image files that are created for each keyword. If the 
user wants to provide an attribute to a keyword, he keys in the attribute and 
selects an input command, using the same operation as for the scene attribute edit 
panel. In addition, when a keyword image is selected from a video image frame, the 
user inputs the commands for capturing and registering the frame functioning as the 
keyword by using the same operation as for the representative frame change panel. 
The editor executes those commands to edit the files 25 and 26. 

Detailed Description Text (96) : 

In the cross -pointing mode, the cross editor updates the hierarchy chart 31 and 
edits the scene information file 21 in response to direct manipulation of an object 
in the keyword chart 32, or conversely, it updates the keyword chart 32 and edits 
the scene information file 21 in response to direct manipulation of an object on the 
hierarchy chart 31. For instance, the manipulation shown by a downward arrow in FIG. 
26 is an instance of the former operation, in which the specification of the 
hierarchy bar 65X in the hierarchy chart 31 is accepted as the specification of the 
keyword bar 66X. That is, if the user specifies the hierarchy bar 65X in the 
hierarchy chart 31 after specifying the boxes 3 8A and 3 8B in the keyword chart 32, 
the cross editor displays the keyword bar 66X in the keyword chart 32 and updates 
the scene information file 21 (c) accordingly. Alternately, the user may directly 
drag the bar 65X to draw the bar 66X. Further, the manipulation shown by an upward 
arrow in FIG. 26 is an instance of the latter operation, in which the specification 
of the keyword bar 66Y is accepted as a command for splitting a scene at the 
starting frame SY and ending frame EY, respectively, of the corresponding interval 
in the hierarchy chart 31. 

Detailed Description Text (97) : 

In the interlocking mode, the cross editor simultaneously updates scene hierarchy 
information and keyword assignment information in response to manipulation of an 
object on either chart. For instance, in the editing keyword chart 32, it updates 
the hierarchy chart 31 so that a scene is split at the' starting and ending frames of 
an interval that was assigned a keyword, interlocking with the manipulation of an 
object on the keyword chart 32, and thereby updating parts (a), (b) , and (c) of the 
scene information file 21. 

Detailed Description Text (98) : 

Thus, during the inputting of scene information, a user interface is provided for 
visually displaying scene information in the form of the hierarchy chart and k'eyword 
chart along with the still image of the representative frame, so that the user can 
grasp scene information intuitively and any erroneous input is immediately fed back. 
In addition, by means of an interface for directly manipulating an object on the 
displayed chart, the frequency of key-ins and erroneous operations at the time of 
input is decreased, as a result of which the input operability is increased. In 
command input operations for verification and correction of scene change points, 
changes of the scene hierarchy, and inputting a scene keyword, the work can be 
clearly seen thanks to the interface for cross-referring to scene information. In 
addition, by using an interface that interprets one command input operation as an 
edit command to different kinds of scene information files, the user can save time 
and labor in checking the consistency of data in those files, and the burden on the 
user of verifying input information is lightened. 

Detailed Description Text (100) : 

In order to perform motion image retrieval by using the still image of a 
representative frame as in the above-identified prior patent application No. 
4-21443, information for accessing a still image file 86 is passed to the unit 23. A 
command for loading the still image file is also generated by using that information 
and transferred to the unit 2 3 for execution. Alternatively, in order to perform 
motion image retrieval by using keywords as in the above prior patent application 
No. 4-65620, it is convenient if the motion image database 24 has keyword* 
information and keyword images. In this case, commands for loading the files 25 and 
26 are generated in a manner similar to that for loading the scene information file 
21 and the representative frame file 22, and sent to the unit 23. 
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CLAIMS : 

1. A computer system for inputting information on scenes that are portions of a 
motion image, said motion image consisting of a linear sequence of frames and 
scenes, each said scene being a continuous sequence of said frames, comprising: 

(a) means for storing motion image data, 

(b) input means, 

(c) display means having a display surface, 

(d) database means, 

(e) means for detecting scene changes in motion image data extracted from said means 
(a) , 

(f) means for generating scene information according to the detected scene changes 
and storing said scene information in a scene information file, 

(g) means for causing graphical display of scene information on the display surface 
of said display means, 

(h) means for executing an edit command input by a user operating said input means 
to edit said scene information file, at least some of said edits serving to apply to 
said scenes in said information file attributes of semantic scene contents, each 
said attribute applying to a continuous sequence of one or more said scenes, and 

(i) means for generating a command executable by said database means for registering 
scene information in said scene information file, and sending the generated command 
to said database means. 

2. In a motion image processing system comprising, in combination, means for storing 
motion image data, input means, display means having a display surface, and database 
means, a method for providing information on scenes that are portions of a motion 
image and registering said information in a scene information file, said motion 
image consisting of a linear sequence of frames and scenes, each said scene being a 
continuous sequence of said frames, said method comprising the steps of: 

(a) extracting motion image data from said means for storing motion image data in 
order to detect scene changes, 

(b) generating scene information according to the detected scene changes and storing 
said scene information in a scene information file, 

(c) displaying objects representing scene information on the display surface of said 
display means, 

(d) interpreting the manipulation of said objects performed by a user operating said 
input means, in order to update said scene information file, some of said 
manipulations serving to apply to said scenes in said information file attributes of 
semantic scene contents, each said attribute applying to a continuous sequence of 
one or more said scenes, and 

(e) registering the scene information contained in said scene information file in 
the database. 

3 . A computer system for inputting information on scenes that are portions of a 
motion image, said motion image consisting of a linear sequence of frames and 
scenes, each said scene being a continuous sequence of said frames, comprising: 

(a) data storage means, 

(b) a representative frame file stored in said storage means for storing still image 
data of representative frames, at least one representative frame being associated 
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with each scene of said motion image, 

(c) a scene information file stored in said storage means for storing the 
information on said scenes, 

(d) input means, 

(e) display means having a display surface 

(f) means for extracting from said representative frame file still image data of 
representative frames for a plurality of scenes, and causing display of the still 
images of the representative frames in a first area of the display surface of said 
display means, said still images being arranged linearly, 

(g) means for extracting from said scene information file scene information related 
to said plurality of scenes, and causing display of said scene information in a 
second area of said display surface, and 

(h) means for executing an edit command input by a user operating said input means, 
thereby causing the editing of said scene information file, at least some of said 
editing serving to apply to said scenes in said scene information file attributes of 
semantic scene contents, each said attribute applying to a continuous sequence of 
one or more said scenes, 

wherein said scene information is information indicating a hierarchical relationship 
of said linear sequence of scenes, and said means (g) causes graphical display of 
said hierarchical relationship of scenes. 

4. A computer system for inputting information on scenes that are portions of a 
motion image, comprising: 

(a) data storage means, 

(b) a representative frame file stored in said storage means for storing still image 
data of representative frames, at least one representative frame being associated 
with each scene of said motion image, 

(c) a scene information file stored in said storage means for storing the 
information on said scenes, 

(d) input means, 

(e) display means having a display surface, 

(f) means for extracting from said representative frame file still image data of 
representative frames for a plurality of scenes, and causing display of the still 
images of the representative frames in a first area of the display surface of said 
display means, said still images being arranged linearly, 

(g) means for extracting from said scene information file scene information related 
to said plurality of scenes, and causing display of said scene information in a 
second area of said display surface, and 

(h) means for executing an edit command input by a user operating said input means, 
thereby causing the editing of said scene information file, 

wherein said scene information is information indicating keywords allocated to 
scenes by a keyword allocation means, and said means (f) causes display of a time 
axis area along the sequence of still images of said representative frames in said 
first area, and said means (g) causes graphical display of the keywords that are 
allocated to scenes corresponding to segments of said time axis. 

5. A computer system as set forth in claim 4 further comprising: 

(i) a keyword information file stored in said data storing means for storing 
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information on keywords allocated to scenes by a keyword allocation means, 

(k) means for retrieving information on said keywords from said keyword information 
file, and causing display of said information in the second area of said display 
surface, and 

(1) means for executing an edit command input by the user operating said input means 
to edit said scene information file, wherein a mode in which said means (g) and (h) 
operate and a mode in which said means (k) and (1) operate change in response to 
specification by the user operating said input means. 
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DOCUMENT- IDENTIFIER: US 5956453 A 

TITLE: Method of editing moving image and apparatus of editing the same 
Abstract Text (1) : 

A method of editing an image with the aid of a computer while watching moving image 
information displayed on a screen includes a step of selecting still images 
representing a plurality of video scenes or a plurality of video cuts constituting 
the moving image information, a step of determining a hierarchical structure among a 
plurality of scenes and a plurality of cuts represented by still images, a step of 
designating static images related to an optional hierarchical structure portion in 
the hierarchical structure, and a step of displaying the hierarchical portion 
including designated still images on a screen with such an arrangement that shows 
the hierarchical structure . An apparatus for editing a moving image for editing an 
image with the aid of a computer while watching moving image information displayed 
on a screen includes a recording apparatus for recording moving image information, a 
change point detection segment for comparing adjacent frame images with each other, 
and recognizing that, when information is changed exceeding a predetermined quantity 
between adjacent frame images, the adjacent frame images are frame images belonging 
to different cuts, reduced images obtained from frame images constituting respective 
cuts and information for identifying the cuts being recorded in the recording 
apparatus, a control segment for generating information for identifying scene images 
by putting a plurality of predetermined cuts together as scene images, generating 
information for identifying a motion picture by collecting a plurality of 
predetermined scenes as a motion picture, and determining a hierarchical structure 
among the scenes and cuts represented by said reduced images in accordance with the 
selection of reduced images representing a plurality of video scenes or a plurality 
of video cuts constituting moving image information , a designation segment for 
designating reduced images related to a predetermined hierarchical structure portion 
in the hierarchical structure, and a display monitor for showing a hierarchical 
portion including designated reduced images with such an arrangement as to show the 
hierarchical structure . 

Brief Summary Text (3) : 

In JP-A-4 -207877, a moving image management apparatus for editing a moving image in 
the unit of scene or cut is disclosed. In this apparatus, a hierarchical structure 
of cuts and scenes is displayed on a screen in a tree structure form. 

Brief Summary Text (4) : 

When edit of a plurality of cuts is proceeded using a tree structure (a hierarchical 
structure ) , however, all of edit states become no longer displayed completely due to 
the limit in point of resolution of a monitor screen. In this case, a user has to 
confirm the display contents by scrolling them, thus causing such a drawback that 
the operation becomes complicated and the contents are difficult to be grasped. 

Brief Summary Text (6) : 

According to the present invention, such a drawback that, when, for example a user 
confirms the whole edit state in case moving images are stored in a tree structure 
(a hierarchical structure ) of a plurality of scenes and cuts, managed and displayed 
as described above in order to search for required scenes and cuts easily, the user 
has to confirm while operating a scroll bar of a window displaying the tree 
structure with a mouse is removed or decreased. 



Brief Summary Text (7) : 

A method of editing a moving image with the aid of a computer while watching moving 
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image information displayed on a screen includes a step in which a plurality of 
reduced still images representing a video scene or a video cut constituting moving 
image information is selected, a step in which a hierarchical structure among still 
images is determined, a step in which still images related to an predetermined 
hierarchical structure portion in the hierarchical structure is designated, and a 
step in which a hierarchical portion including the designated still images is 
displayed with such a hierarchical arrangement. 

Brief Summary Text (8) : 

A recording medium for storing program codes read and executed by a computer 
according to the present invention stores a first code section representing a 
procedure for determining a hierarchical structure among the still images in 
accordance with the selection of a still image representing a video scene or a video 
cut constituting moving image information, and a second code section representing a 
procedure for displaying with such an arrangement that a hierarchical portion 
including designated still images that expresses the hierarchical structure in 
accordance with that still images related to a predetermined hierarchical structure 
portion in the hierarchical structure are designated. 

Brief Summary Text (9) : 

An apparatus for editing a moving image with the aid of a computer while watching 
moving image information displayed on a screen according to the present invention 
includes a recording apparatus for recording the moving image information, a change 
point detection segment for detecting a change point of a moving image comparing 
adjacent frame images with each other and recognizing that the adjacent frame images 
are frame images belonging to different cuts when the information changes beyond a 
predetermined quantity of the information difference between adjacent frame images, 
a recording segment for recording reduced still images obtained by reducing data 
quantity of frame images forming respective cuts and information or reduced image 
display file for indicating reduced still images representing the cuts, a control 
segment for generating information for identifying a scene by putting a plurality of 
predetermined cuts together as a scene, generating information for identifying a 
moving image by putting a plurality of predetermined scenes together as a moving 
image or a motion picture, and determining a hierarchical structure among image 
scenes and image cuts represented by reduced still images in accordance with the 
selection of a reduced still image representing a scene or a video cut constituting 
moving image information, a designating segment for designating reduced still images 
related to an predetermined hierarchical structure portion in the hierarchical 
structure, and a monitor for. making graphic expression with such an arrangement that 
the hierarchical portion including the designated reduced still images show the 
hierarchical structure . 

Drawing Description Text (7) : 

FIGS. 6A, 6B and 6C are diagrams for explaining examples of hierarchical structure 
display of moving image information by a combination of representative images; 

Drawing Description Text (8) : 

FIGS. 7A and 7B show a flow chart of processing in case only a part of a 
hierarchical structure is displayed on a monitor screen in an editing method of an 
embodiment of the present invention; 

Detailed Description Text (3) : 

In FIG. 8, a frame image array constituting a recorded video source is displayed 
typically as a time series arrangement. The image data of a video source recorded on 
a magnetic tape are generally arranged in such a form. In a magnetic disk and an 
optical disk, the image data are not necessarily recorded in such a time series 
arrangement physically, but another data arrangement is made. Since it is possible 
to make access to any recorded image data in almost the same time in a disk-shaped 
recording medium, such recording medium is advantageous. 

Detailed Description Text (4) : 

A "cut" is defined as a set of frame images picked up from the start to the end of 
photographing operation one time with a TV camera. A "scene" is composed of a set of 
a plurality of cuts in which a predetermined object is picked up. It is possible to 
put a plurality of scenes together under a specific image pickup theme. The video 
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source can be arranged as such a hierarchical structure of information as a 
plurality of cuts, scenes putting several cuts together, and a motion picture having 
an image pickup theme (or a title) putting several scenes together. A specifying 
number is assigned to the cut and the scene, respectively. The number of the layers 
in the hierarchical structure may be increased by extending the concept of a scene 
with using a subset such as a chapter. 

Detailed Description Text (5) : 

FIG. 9 shows an example of a tree -form hierarchical structure of a recorded image 
source. The image pickup theme is "Tropical Island". The recorded image source 
includes a Cut 1 in which fishes in the sea around the island are picked up, a Cut 2 
in which a coastal landscape is picked up, a Cut 3 in which the reptiles inhabiting 
a forest are picked up, a Cut 4 in which flowers are picked up, a Cut 5 in which 
birds are picked up, a Cut 6 in which a market view is picked up and a Cut 7 in 
which a harbor is picked up. The Cut 1 is composed of a first frame to a 90th frame 
for instance, and the Cut 2 is composed of a 91st frame to a 150th frame. The images 
of the Cut 1 and the Cut 2 are put together as a Scene 1 in which the picked up 
object is an image of sea. The Cuts 3, 4 and 5 are put together as a Scene 2 in 
which the picked up object is a forest. The Cuts 6 and 7 are put together as a Scene 
3 in which the picked up object is a town view. 

Detailed Description Text (6) : 

In an editing apparatus according to an embodiment of the present invention which 
will be described in detail later, it is possible to display a part or the whole of 
this hierarchical structure on a monitor screen for editing. It is possible to 
optionally designate which part of the hierarchical structure is to be displayed . On 
the screen to display the hierarchical' structure, a representative frame image among 
those frame images that constitute respective cuts and a representative frame image 
among those frame images that constitute respective scenes are displayed as a 
reduced still image, respectively. It is possible to display not only a reduced 
still image, but also character information showing an attribute of a reduced still 
image on a screen to display a hierarchical structure . 

Detailed Description Text (7) : 

According to the embodiment of the present invention, editing of a moving image can 
be made easily by changing an original hierarchical structure shown on a screen by a. 
simple operation by an operator. For example, it is possible to change, i.e., to 
edit the Cut 1, the Cut 3 and the Cut 5 to a hierarchical structure having a lower 
hierarchy of a new Scene 1 with animals living in the island as the picked up 
object. When an image in which the hierarchical structure is changed in such a way 
is reproduced, a motion picture edited in the order of the Cut 1, the Cut 3 and the 
Cut 5 can be seen. In the editing work using a screen to display a hierarchical 
structure, replacement or elimination of the cut or addition of another cut and so 
on is possible easily. Furthermore, the sequence of scenes can also be replaced 
easily. For example, it is possible to edit so as to constitute a sequence such as 
the Scene 3, the Scene 1 and the Scene 2. In that case, the sequence of the cuts of 
the lower hierarchy of a scene is also replaced together. 

Detailed Description Text (8) : 

FIG. 1 shows a monitor screen 200 of an apparatus for editing a moving image 
according to an embodiment of the present invention. In the figure, a window 201 
represents an area for editing. Representative images of cuts 210 to 214 and scenes 
220 to 221 constituting moving images of the video source to be edited are displayed 
on the area for editing 201 in a scale-down size. These scale-down images or reduced 
images are displayed in a tree -form hierarchical structure . 

Detailed Description Text (10) : 

FIG. 2 is a block diagram of a structural example of an apparatus for editing a 
moving image. Namely, there are provided a CPU 101 for processing various controls, 
a monitor 108 for displaying information of a hierarchical structure (a tree 
structure ) , showing scenes and cuts used for editing a moving image and indicating 
an edited state and so on, a memory 102 for storing various control programs of the 
CPU 101 and so on, a frame buffer 107 for storing images to be displayed on a 
screen, and a cut change point detection segment 103 for detecting a change point of 
a series of moving images composed of a plurality of cuts as an input. There is 
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further provided a magnetic storage device 106 for storing a series of moving 
images, moving image relational information consisting of a frame number and a 
reduced image display file name in which change points detected in the cut change 
point detection segment 103 are described, a reduced moving image for high speed 
display for watching the whole moving image and the scene and so on which are 
consisted by these plurality of frame images without stretching processing and also 
reduced to the size displayed as representative image and hierarchical structure 
management information consisting of a hierarchical number showing what number of 
the hierarchy each of the plurality of scenes or cuts is located, a reduced image 
display file name, a cut number attached serially from a first cut for each cut 
consisting of a plurality of frames, and identifiers showing whether to display on a 
monitor 108 displaying a hierarchical structure or not. Furthermore, there are 
provided a video interface 104 for receiving a video signal from a VTR 105 and 
converting it into a format for handling a moving image with this apparatus, a mouse 
or a keyboard 109 as input means, and an image compressing segment 112 for 
compressing the image information inputted from the VTR 105 via the video interface 
104 to produce the compressed image data, and expanding the compressed image data to 
reproduce the image information. 

Detailed Description Text (11) : 

The above-mentioned CPU 101 is connected to a bus 110. And memory 102, cut change 
point detection segment 103, video interface 104, magnetic storage device 106, frame 
buffer 107 and image compressing segment 112 are also connected to a bus 110, and 
are controlled by the access from the CPU 101. The other storage medium may be used 
or a remote file through a network (such as a network using LAN, RS-232C interface) 
may be used in place of the magnetic storage device 106. 

Detailed Description Text (14) : 

When a cut change point is decided and the sectioning of the cuts is determined, 
data of moving image relational information files corresponding to each cut at 
sectioned with the cut change point as shown in FIG. 4 are generated and stored in 
the magnetic storage device 106. In a cut number slot 301 of the moving image 
relational information file, the cut number of that cut is held, in a cut length 
slot 302, a value corresponding to the number of frames included in the cut is held, 
in a reduced image display file name slot 303, an almost peculiar cut title 
corresponding to the cut is held, and, in a frame number slot 304, the frame number 
of a frame image, for example at the head of the cut is held. In this case, since 
the whole moving image or the scene consisting of a plurality of frames is seen at a 
high speed, reduced images are stored in the magnetic storage device 106 as a moving 
image reduced to the size displayed at a high speed as a high speed display image 
and displayed as a representative image. t 

Detailed Description Text (15) : 

In order to take out a predetermined frame image from coded moving image 
information, decoding and expansion are required, which requires processing time 
with the CPU 101. However, when uncoded moving image information which is image 
information reduced for high-speed display is stored in advance as the present 
embodiment, it is possible to read out the frame image without processing time 
required for decoding. 

Detailed Description Text (16) : 

An operator instructs to read out a moving image registered in the magnetic storage 
device 106 by means of a mouse or a keyboard 109. Then, the CPU 101 reads only the 
first frame of each cut as the change point of the cut of the corresponding moving 
image, 'and displays the cut table window 203 of a screen 200 displayed in a monitor 
108 as shown in FIG. 1. Corresponding to this display, the operator decides whether 
the change point of the cut is correct or not. The operator now indicates delimiter 
of the cuts and now changes the delimiter of the cut by means of the mouse or the 
keyboard 109. 

Detailed Description Text (21) : 

When the editing is started and an operator selects a reduced still image 804 of the 
Cut n and a reduced still image 8 05 of the Cut j displayed in the raw material area 
203 by means of the mouse or the keyboard 109 and clicks a "junction" key 810 
selectively when these images are in a selected state, reduced still images 804' and 
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805' corresponding to the Cut n and the Cut j , respectively, are newly displayed as 
shown in FIG. 6A in the editing area 201 where reduced still images have not been 
displayed until starting editing. Furthermore, a reduced still image 806 
corresponding to a scene (a Scene k) composed of the Cut n and the Cut j is 
displayed, a connection made to show a related state of these three reduced still 
images 804', 805' and 806 is displayed at the same time, and a diagram showing a 
tree structure thereof can be displayed quickly with simple operationality . 

Detailed Description Text (22) : 

At this time, a hierarchical structure management information file consisting of a 
picture management information file, a scene management information file and a cut 
image information file corresponding to newly displayed tree structure display 
described above such as shown in FIG. 5 is generated. In this case, the number of 
the hierarchy where the Scene k is positioned such as "1" is held in a hierarchical 
number slot 321 among the slots of the scene management information file, a scene 
number corresponding to the Scene k such as "k" is held in a scene number slot 322, 
and one of the cuts which becomes a member positioned in the lower hierarchy of the 
Scene k such as a cut number "n" corresponding to the Cut n is held in a cut number 
1 slot 324 which becomes a member. Similarly, a cut which becomes a member 
positioned in the lower hierarchy of the Scene k such as a cut number "j" 
corresponding to the Cut j is held in a cut number 2 slot 325 which becomes a 
member. In this case, since the number of cuts positioned in the lower hierarchy of 
the Scene k is 2, data are not held in a cut number 3 slot 326 which becomes a 
member, but the slot 326 is empty. Furthermore, coordinates information for 
determining a display position of the reduced still image 806 corresponding to the 
Scene k and in a displaying position in accordance with the editing area 201 is held 
in an icon display coordinates slot 327, and information for identifying whether the 
reduced still image 806 is to be displayed in the editing area 201 or not such as 
information "to display " is held in a display identifier slot 328. 

Detailed Description Text (23) : 

Furthermore, cut image information files corresponding to the Cut n and the Cut j 
are generated as a cut image information file among hierarchical structure 
management information files. Among the slots of cut image information files 
corresponding to the Cut n, the number of the hierarchy where the Cut n is 
positioned such as "2" is held in a hierarchical number slot 331, a cut number 
corresponding to the Cut n such as "n" is held in a cut number slot 301', and a 
value corresponding to the number of frames of the Cut n is held in a cut length 
slot 302'. Furthermore, an almost peculiar file title corresponding to the contents 
of the Cut n is held in reduced image display file name slot 303', and a frame 
number of a frame image, for example at the head of the Cut n is held in a frame 
number slot 304'. Furthermore, coordinates information for determining a display 
position of a reduced still image 804 1 corresponding to the Cut n in accordance with 
the editing area 201 is held in an icon display coordinates slot 332, and 
information for identifying whether the reduced still image 804* is to be displayed 
in the editing area 201 or not such as information "to display " is held in a display 
identifier slot 333. 

Detailed Description Text (25) : 

Furthermore, a motion picture image constituted from a plurality of scenes displayed 
in the editing area 201 such as a motion picture 1 is generated as an upper 
hierarchy of those respective scenes, and reduced still image corresponding to the 
motion picture 1 is connected to reduced still images corresponding to respective 
scenes and displayed . The state of the tree structure is shown in FIG. 6B. A 
hierarchical structure management information file such as shown in FIG. 5 described 
above is generated in accordance with the hierarchical structure display . The number 
of the hierarchy where the motion picture 1 is positioned such as "1" is held in a 
hierarchical number slot 311 among the slots in the picture management information 
file of the hierarchical structure management information file. 

Detailed Description Text (27) : 

Furthermore, a motion picture number corresponding to the motion picture 1 such as 
"1" is held in a motion picture number slot 312 among the slots in the picture 
management information file. Furthermore, a scene number corresponding to one of the 
scenes that become members located in the lower hierarchy of the motion picture 1 
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such as a scene number "1" corresponding to the Scene 1 is held in a scene number 1 
slot 313 which becomes a member. Similarly, a scene number corresponding to a scene 
which becomes a member located in the lower hierarchy of the motion picture 1 such 
as a scene number "2" corresponding to the Scene 2 is held in a scene number 2 slot 
314 which becomes a member, and a scene number corresponding to a scene which 
becomes a member located in the lower hierarchy of the motion picture 1 such as a 
scene number "3" corresponding to the Scene 3 is held in a scene number 3 slot 315 
which becomes a member. Furthermore, coordinates information for determining a 
display position of a reduced still image 601 corresponding to the motion picture 1 
in accordance with the editing area 2 01 are held in an icon display coordinates slot 
316, and information for identifying whether the reduced still image 601 is to be 
displayed in the editing area 201 or not such as information "to display " is held in 
a display identifier slot 317. 

Detailed Description Text (28) : 

As described above, a plurality of reduced still images corresponding to scenes, 
cuts and the motion picture in respective hierarchies are displayed in the editing 
area 201 as shown in FIG. 6B. Since the size of the editing area 201 which can be 
displayed is restricted, however, it is conceived to display only the cuts in the 
lower hierarchy of the selected scene, and selective display switching is realized 
by means of the operation described hereunder in the present invention. 

Detailed Description Text (29) : 

Namely, an operator selects one of the reduced still images corresponding to 
respective scenes displayed in the editing area 201 with the mouse or the keyboard 
109. When a " display " key 811 is clicked selectively in case the reduced still image 
is in a selection state, reduced still images corresponding to the cuts positioned 
in the lower hierarchy of the scenes where the reduced still images are selected are 
displayed as they are, and reduced still images corresponding to the cuts positioned 
in the lower hierarchy of the scenes other than the scenes where the reduced still 
images are selected become no longer displayed . A change-over of setting whether the 
reduced still image is to be displayed or not is performed by rewriting the held 
information in display identifier slots of respective cut image information files 
corresponding to the cuts not to display in the above -explained hierarchical 
structure management information file from a value "to display " to a value "not to 
display " . It is possible to be performed by rewriting the held information for not 
only cuts but also scenes. 

Detailed Description Text (32) : 

When the " display " key 811 is clicked selectively in case the reduced still image 
601 is in a selected state, as to the reduced still image displayed in the editing 
area 201, the reduced still images corresponding to the motion picture 1, the Scene 
1, the Scene 2 and the Scene 3 displayed in FIG. 6B are kept as they are displayed, 
and similarly, the reduced still images corresponding to the Cut 1, the Cut 2 and 
the Cut 3 positioned in the lower hierarchy of the Scene 1 are kept as they are 
displayed . The reduced still images corresponding to the other Cut 4, Cut 5 and Cut 
6 that have been displayed in FIG. 6B are not displayed in FIG. 6C. 

Detailed Description Text (33) : 

In this case, scene management information files of the scenes other than the 
selected scene, i.e., the scenes other than the Scene 1 are referred to sequentially 
using the hierarchical structure management information file. For example, 
information held in the cut number 1 slot 323 which becomes a member and in the cut 
number 2 slot 324 which becomes a member in the scene management information file of 
the Scene 2, i.e., the cut numbers of the Cut 4 and the Cut 5 are read out 
respectively. Further, the cut image information file of the cuts in the lower 
hierarchy of the Scene 2 is referred to on the basis of the cut numbers which have 
been held in respective slots and read out therefrom, and the values of respective 
display identifier slots of the cut image information file corresponding to the Cut 
4 and the cut image information file corresponding to the Cut 5 are set to "not to 
display " . Similarly, the similar processing to the above is performed with respect 
to the scene management information file of the Scene 3. 

Detailed Description Text (34) : 

Although the tree hierarchical structure has been described as a three -layer 
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structure in the above description, it is possible to display only the reduced still 
images at the selected reduced still image or thereafter among a plurality of 
reduced still images at a desired level by setting the values of the display 
identifier slots of the lower hierarchy with similar operation even when the number 
of layers is more than three layers. 

Detailed Description Text (36) : 

The changing of the display position of the reduced still image is performed by 
rewriting the held information in the icon display coordinates slots of respective 
scene management information file or the cut image information file corresponding to 
the objective scenes or cuts in the above-mentioned hierarchical structure 
management information file into the value of a predetermined coordinate 
information . 

Detailed Description Text (38) : 

The operator selects a reduced still image 601 corresponding to Scene 1 displayed in 
the editing area 201 shown in FIG. 6B with operating the mouse or the keyboard 109. 
When the "display" key 811 is clicked selectively in a selected state, as to the 
reduced still image displayed in the editing area 201, the reduced still images 
corresponding to the motion picture 1, Scene 1, Scene 2 and Scene 3 displayed in the 
screen of FIG. 6B are kept as they are displayed, and the displayed positions are 
changed, that is for example the distances between adjacent reduced still images 
corresponding to the Scene 1, Scene 2 and Scene 3 on the screen are shortened. 
Further, the reduced still images corresponding to the Cut 1, Cut 2, Cut 3, . . . 
and Cut i positioned in the lower hierarchy of the Scene 1 are displayed at the 
changed positions. 

Detailed Description Text (39) : 

In this case, the scene management information files of the scenes whose 
corresponding reduced still images are changed in the position on the screen and the 
cut image information files of the cuts whose corresponding reduced still images are 
changed in the position on the screen are referred to by using the hierarchical 
structure management file. And then, the values of the icon display coordinates slot 
in the scene management information file of e.g. the Scene 2 is set to a value which 
makes the distance between the adjacent reduced still images corresponding to the 
Scene 1 and Scene 2 shorten. The same process is also taken for the values of the 
icon display coordinates slot in the scene management information file related to 
the other scenes or the cut image information file related to the cuts. 

Detailed Description Text (40) : 

A processing flow chart when only a part of hierarchical structure designated by an 
operator is displayed is shown in FIGS. 7A and 7B. 

Detailed Description Text (41) : 

In a step 701, a reduced still image corresponding to a scene is selected. In a step 
702, the "display" key 811 in an icon 240 is clicked. In a step 703, access is made 
to the picture management information file (FIG. 5- (A) ) in accordance with a motion 
picture in the upper hierarchy of the scene (selected scene) corresponding to the 
selected reduced still images. In a step 704, "a scene number which becomes a 
member" corresponding to other „than the selected scenes in the file is referred to. 
In a step 705, it is determined whether the slot of "the scene number which becomes 
a member" of the picture management information file is empty or not. If the slot is 
empty, the processing is completed in a step 706. If the slot is not empty, access 
is made to the scene management information file (FIG. 5-(B)) of the scene in 
accordance with the data contents of "the scene number which becomes a member" in a 
step 707. In a step 708, a slot of the first "cut number which becomes a member" of 
the scene management information file is selected. In a step 709, access is made to 
the cut image information file (FIG. 5- (C) ) of the cut in accordance with the data 
contents of the slot of "the cut number which becomes a member". In a step 710, the 
data contents of the slot of the display identifier are set to "not to display " . In 
a step 711, it is determined whether the slot of the second "cut number which 
becomes a member" in the scene management information file (FIG. 5- (B) ) is empty or 
not. If the slot is not empty, the processing is returned to the step 709. In a step 
712, it is determined whether the slot of the second "scene number which becomes a 
member" of the picture management information file (FIG. 5- (A)) is empty or not. If 
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the slot is empty, the processing is completed, and if the slot is not empty, the 
processing is returned to the step 707. 

Detailed Description Text (42) : 

As described above, according to the present invention, it is possible to grasp the 
edited video contents quickly by arranging so that, when reduced still images 
corresponding to scenes and cuts constituting a motion picture are displayed in 
accordance with a hierarchical structure, all of representative images of the whole 
hierarchical structure are not displayed with scrolling, but only reduced still 
images corresponding to the scenes or the cuts that belong to the scenes of the 
selected hierarchy or thereafter are displayed with respect to the scenes and the 
cuts in a predetermined hierarchy or thereafter. 

CLAIMS : 

1. A method of editing moving images comprising: 

inputting said moving images, said moving images including a plurality of scenes, 
each scene including a plurality of cuts and each cut including a plurality of 
frames ; 

detecting change portions of said moving images and dividing said moving images into 
said scenes and cuts; 

selecting a still image representing each of said scenes and cuts; 

storing position information with respect to said change portions of said moving 
images and said representative still images of said scenes and cuts; 

designating said representative still images of said scenes and cuts to edit said 
moving images ; 

determining a hierarchical structure, said hierarchical structure including at least 
a plurality of said representative still images of said scenes and said 
representative still images of said cuts relating to at least one scene selected 
from said scenes; and 

displaying an entire image of said hierarchical structure on a screen, whereby said 
hierarchical structure is changed in accordance with a requirement of an operator. 

2. A method of editing moving images according to claim 1, wherein said step for 
determining said hierarchical structure has a step for generating a hierarchical 
structure management information file including display identification information 
for specifying representative still images to be displayed on said screen and 
information for designating a hierarchical relationship between said cuts and scenes 
constituting said moving images, and, in said display step, designated 
representative still images are displayed on a screen as a hierarchical structure 
based on the information of said hierarchical structure management information file 
including said display identification information . 

3. A method of editing moving images according to claim 2, further comprising: 

changing the hierarchical structure management information file by designating the 
representative still images on said screen showing said hierarchical structure and 
displaying the changed hierarchical structure on a screen. 

4. A method of editing moving images according to claim 1, wherein said hierarchical 
structure further includes a plurality of frames relating to said selected cuts. 

5. A method of editing moving images according to claim 4, further comprising: 

changing coordinate positions on said screen of said representative still images 
displayed with said hierarchical structure so that said representative still images 
and said frames are displayed within a predetermined screen area. 
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6. A method of editing moving images according to claim 4, wherein in said step of 
displaying said hierarchical structure including said designated representative 
still images, both or either of the image representative frame in the frame images 
constituting said cuts and scenes and character information showing the contents of 
said cuts and scenes is displayed in said hierarchical structure . 

7. A recording medium for storing program codes read and executed by a computer, 
comprising : 

a first code section for inputting said moving images, said moving images including 
a plurality of scenes, each scene including a plurality of cuts and each cut 
including a plurality of frames ; 

a second code section for detecting change portions of said moving images and 
dividing said moving images into said scenes and cuts; 

a third code section for selecting a still image representing each of said scenes 
and cuts; 

a fourth code section for storing position information with respect to said change 
portions of said moving images and said representative still images of said scenes 
and cuts; 

a fifth code section for designating said representative still images of said scenes 
and cuts to edit said moving images; 

a sixth code section for determining a hierarchical structure, said hierarchical 
structure including at least a plurality of said representative still images of said 
scenes and said representative still images of said cuts relating to at least one 
scene selected from said scenes; and 

a seventh code section for displaying an entire image of said hierarchical structure 
on a screen, whereby said hierarchical structure is changed in accordance with a 
requirement of an operator. 

8. An apparatus for editing moving images comprising: 

input means for inputting said moving images, said moving images including a 
plurality of scenes, each scene including a plurality of cuts and each cut including 
a plurality of frames ; 

detecting means for detecting change portions of said moving images and dividing 
said moving images into said scenes and cuts; 

selecting means for selecting a still image representing each of said scenes and 
cuts; 

a memory for storing position information with respect to said change portions of 
said moving images and said representative still images of said scenes and cuts; 

means for designating said representative still images of said scenes and cuts to 
edit said moving images; 

means for determining a hierarchical structure, said hierarchical structure 
including at least a plurality of said representative still images of said scenes 
and said representative still images of said cuts relating to at least one scene 
selected from said scenes; 

a display for displaying an entire image of said hierarchical structure on a screen; 
and 

a controller for controlling said detection means, said selecting means, said 
memory, said designating means, said determining means and said display so that said 
hierarchical structure is changed in accordance with a requirement of an operator. 
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9. A method of editing moving images comprising: 

compressing said moving images and storing compressed moving images into a storage 
device, said moving images including a plurality of scenes, each scene including a 
plurality of cuts and each cut comprising a plurality of frames ; 

detecting change portions of said moving images and dividing said moving images and 
dividing said moving images into said scenes and cuts; 

selecting a still image representing each of said scenes and cuts; 

storing position information with respect to said change portions of said moving 
images and said representative still images of said scenes and cuts; 

designating said representative still images of said scenes and cuts to edit said 
moving images and storing said representative still image of each of said scenes and 
cuts in a form of a reduced still image; 

determining a hierarchical structure, said hierarchical structure including at least 
a plurality of said representative reduced still images of said scenes and said 
representative reduced still images of said cuts relating to at least one said scene 
selected from said scenes; and 

displaying said reduced still images representing said scenes and cuts designated to 
edit said moving images in the form of an entire image of said hierarchical 
structure, wherein said hierarchical structure is changed in accordance with a 
requirement of an operator. 

10. A method of editing moving images according to claim 9, wherein when an operator 
selects one of said representative reduced still images of said hierarchical 
structure, lower levels of said hierarchical structure related to said selected 
representative reduced still image are displayed while said entire image of said 
hierarchical structure is being displayed . 

12. An editing apparatus for editing moving images comprising: 

a video interface through which said moving images are supplied to the editing 
apparatus, said moving images including a plurality of scenes, each scene including 
a plurality of cuts and each cut including a plurality of frames ; 

a cut point detector, coupled to said video interface which detects change portions 
of said moving images, divides said moving images into said scenes and cuts and 
designates a still image representing each of said scenes and cuts; 

a storage device which stores compressed image data corresponding to said moving 
images, change portion information with respect to said change portions of said 
moving images and a designated still image representing each of said scenes and cuts 
in the form of a reduced still image; 

a processor for determining a hierarchical structure, said hierarchical structure 
including at least a plurality of said representative reduced still images of said 
scenes and said representative reduced still images of said cuts relating to at 
least one scene selected from said scenes; 

a monitor which displays an entire image of said hierarchical structure on a screen; 
and 

a control unit which controls said video interface, said cut point detector, said 
storage device, said processor and said monitor so that said hierarchical structure 
is changed in accordance with a requirement of an operator. 

13. An editing apparatus for editing moving images according to claim 12, wherein 
when an operator selects one of said representative still images of said 
hierarchical structure, lower levels of said hierarchical structure related to said 
selected representative still image are displayed while said entire image of said 
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hierarchical structure is being displayed . 



